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ABSTAACT 

A hierarchical arcbitactura is described which supports space station telerobots in a 
variety of nodes. The systen is divided into three hierarchies; task deconposi tion, world 
nodal, and sensory processing. Goals at each level of the task decoaposition hierarchy are 
divided both spatially and teaporally into siapler connands for the next lower level. This 
decoaposition is repeated until, at the lowest level, the drive signals to the robot 
actuators are generated. To accoaplish its goals, task decoaposition nodules nust often use 
inforaation stored in the world nodel. The purpose of the sensory systen is to update the 
world nodel as rapidly as possible to keep the nodel in registration with the physical 
world. £he architecture of the entire control systen hierarchy and how 
it can be applied to space telerobot applications/^ ru d. € i ' . ‘ /' 


1. INTRODUCTION 

One of the najor directions on which the robot research connunity has concentrated its 
efforts is concerned with planning and controlling notion. Given a specific task, a notion 
plan aust be calculated which neets the task requirenents • Then, the plan nust be executed* 
there nust be sufficient control for the robot to adequately effect the desired notion. 

Trajectories are often planned as straight lines in Cartesian space (1). Whitney (2,3) 
developed the resolved notion rate control nethod for Cartesian straight line notions* 
Paul (4,5,6) used honogeneous coordinate transf ornations to describe a trajectory as a 
function of tiae, and Taylor (7) used coordinated joint control over saall segaents to keep 
the trajectory within a specified deviation of the desired straight line trajectory. 

While the research described above enploys a *kineaatic* approach to robot control, 
another direction of research takes the sanipulator "dynamics” into account in the 
description of robot notion. The dynaaic equations of notion are described either by the 
Lagrangian fornulation (8) or by the Newton-Buler equations (9) . Algorithas and coaputer 
architectures have been suggested which proaise real-time dynaaic robot control (10,11). 

Another aspect of notion control is concerned with the variables being controlled. The 
research described to this point was concerned priaarily with position control. The robot 
aoved froa an initial position to a goal position. While this is perhaps the aost coaaon 
node, there are aany applications for robots which suggest that other variables should be 
controlled. For exaaple, force control would be desired for asseably operations, taibert 
and Craig (12) suggest a method for hybrid position/force control of aanipulator s. 

These examples point to the sore general problea of sensory processing. For a great 
deal of robot notion research, sensory processing has been United to joint positions, 
velocities, and accelerations. However, other sensors are often required to accoaplish 
tasks. The control coaaunity has concentrated on the control aspects of the robot and as a 
result, little eaphasis has been placed on sophisticated sensory processing. 

Machine vision, an offshoot of image processing research, has recently been associated 
with advanced robot applications. One of the aost interesting directions in this research 
area is concerned with sensor controlled robots. Operating with the constraints iaposed by 
real-tine robot control, early methods used structured light and binary iaagea 
(13,14,15,16). These approaches, though developed at different institutions, shared aany 
concepts. One of the important subsequent research efforts vent toward the developnent of 
aodel-based inage processing. Bolles and Cain (17) used models of objects to guide the 
algorithas in a hypothesis/verification acheae known as the local feature focus method. Tha 
concept has recently been extended froa two diaensional (i.e. nearly flat) objects to three 
dimensional objects (18). Although the approaches described here have led to a better 
understanding of real-tiae vision processing, the systems lacked a sophisticated 
interconnection with the robot control system. 
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Tb« lutoatttd Manufacturing Research fwiuty (AM BP) * developed at the national Burma* 
of Standards* Is a hierarchically organised snail-batch netal machining shop (19). Zt 
separates sensory processing and robot control by a sophisticated world model. The world 
model has three complementary data representations. Lamia (20) describes the CAD- like 
section of the model. Shaeler, Kent* and Mansbach (21) describe the octree and table 

representations supported by the model. The model generates hypotheses for the features 
which are either aerified or refuted by empirical evidence. The sensory system* s task is to 
update the appropriate parts of the world model with new or revised data as rapidly ** 
possible. The control system accesses the world model as desired to obtain the current best 
guess concerning any aspect of the world. shneier* Lomia* and lent (22) describe the 
sensory system and its operation in greater detail. The AM IIP was the first deliberate 
attempt to tie together sensory processing* world modeling* and robot control in a generic 
fashion. The system developed for the AMRP is applicable to more than manufacturing* This 
paper describes its use in space telerobotics. 

2* A PUBCTIOMAL SYSTEM ARCHITECTURE 

• 

The fundamental paradigm is shown in Figure 1. The control system architecture Is a 
three legged hierarchy of computing modules* serviced by a communications system and a 
common memory. The task decomposition modules perform real-time planning and task 
monitoring functions* and decompose task goals both spatially and temporally. The sensory 
processing modules filter* correlate* detect* and integrate sensory information over both 
space and time in order to recognize and measure patterns* features* objects* events* and 
relationships in the external world. The world modeling modules answer queries* make 
predictions* and compute evaluation functions on the state space defined by the information 
stored in common memory. Common memory is a global database which contains the system's 
best estimate of the state of the external world. The world modeling modules keep the 
common memory database current and consistent. 

2.1. Task Decomposition - H nodules 

(Plan* Execute) 

The first leg of the hierarchy consists of task decomposition H modules which plan and 
execute the decomposition of high level goals into low level actions. Task decomposition 
involves both a temporal decomposition (into sequential actions along the time line) and a 
spatial decomposition (into concurrent actions by different subsystems). Bach R module at 
each level consists of a job assignment manager JA* a set of planners PI«(1)* and a set of 
executors EX(i). These decompose the input task into both spatially and temporally distinct 
subtasks as shown in rigure 2. This will be described in greater detail in section 4. 

2.2. World Modeling - M modules 

(Remember* Estimate* Predict* Evaluate) 

The second leg of the hierarchy consists of world modeling M modules which model (i.e. 
remember* estimate* predict) and evaluate the state of the world. The "world model" is the 
system's best estimate and evaluation of the history* current state* and possible future 
states of the world* including the states of the system being controlled. The "world model" 
includes both the M modules and a knowledge base stored in a common memory database where 
state variables, maps* lists of objects and events* and attributes of objects aad events are 
maintained. By this definition* the world model corresponds to what is widely knows 
throughout the artificial intelligence community as a "blackboard" (23). The world model 
performs the following functions: 

1. Maintain the common memory knowledge base by accepting information from the 
sensory system. 

2. Provide predictions of expected sensory input to the corresponding G modules* 

based on the state of the task and estimates of the external world. 

3. Answer "tfhat is?" questions asked by the executors in the corresponding level I 
modules. The task executor can request the values of any system vsrisble. 

4. Answer "What if?" questions asked by the planners in the corresponding level I 
modules. The M modules predict the results of hypothesised actions. 

2.3. Sensory Processing - 6 modules 

(Filter* Integrate* Detect, Measure) 

The third leg of the hierarchy consists of sensory processing G modules. These 
recognize patterns* detect events* and filter and integrate sensory information over space 
and time. The G modules at each level compare world model predictions with sensory 
observations and compute correlation and difference functions. These are integrated over 
time and space so as to fuse sensory information from sultiple sources over eztended time 
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intervals. Itwly detected or recognised «v«nts, objects, and relationships are entered by 
the N nodules into the world nod el coanon aeaory database, and objects or relationships 
perceived to no longer exist are renoved. The G nodules also contain functions which can 
conpute confidence factors and probabilities of recognised events, and statistical estinates 
of stochastic state variable values. 

2.4. Operator Interfaces 

(Control, Observe, Define Goals, Indicate Objects) 

The control architecture defined here has an operator interface at each level in the 
hierarchy. The operator interface provides a neans by which hunan operators, either in the 
space station or on the ground, can observe and supervise the telerobot. Sach level of the 
task deconposition hierarchy provides an Interface where the hunan operator can assume 
control. The task commands into any level can be derived either from the higher level ■ 
module, or from the operator interface. Osing a variety of input devices such as a 
joystick, mouse, trackball, light pen, keyboard, voice input, etc., a human operator can 
enter the control hierarchy at any level, at any time of his choosing, to monitor a process, 
to insert information, to interrupt automatic operation and take control of the task being 
performed, or to apply human Intelligence to sensory processing or world modeling functions. 

The sharing of command input between human and autonomous control need not be all or 
none. It is possible in many cases for the hunan and the automatic controllers to 
simultaneously share control of a telerobot system. For example a human might control the 
orientation of a camera while the robot automatically translates the same camera through 
space. 

2.4.1 Operator Control interface levels 

The operator can enter the hierarchy at any level. The operator control interface 
interprets teleoperation in the fullest senses a teleoperator is any device which is 
controlled by a human from a remote location. while the master-slave paradigm is certainly 
a type of teleoperation, it does not constitute the only form of man-machine interaction. 
At different levels of the hierarchy, the interface device for the hunan may change but the 
fundamental concept of teleoperation is still preserved. Table 1 illustrates the 
interaction an operator may have at each level. 

The operator control interface thus provides mechanisms for entering new instructions 
or programs into the various control modules. This can oe used on-line for real-time 
supervisory control, or in a background mode for altering autonomous telerobot plans before 
autonomous execution reaches that part of the plan. 

2.4.2 Operator monitoring interfaces 

The operator interfaces allow the human the option of simply monitoring any level. 
Windows into the common memory knowledge base permit viewing of maps of service bay layout, 
geometric descriptions and mechanical and electrical configurations of satellites, lists of 
recognised objects and events, object parameters, and state variables such as positions, 
velocities, forces, confidence levels, tolerances, traces of past history, plans for future 
actions, and current priorities and utility function values. These nay be displayed in 
graphical fora, for example using dials or bar graphs for scalar variables, shaded graphics 
for object geometry, and a variety of map displays for spatial occupancy. 

2.4.3 Sensory pr ocess i ng/wor Id modeling interfaces 

The operator interface may also permit interaction with the sensory processing and/or 
world modeling modules. For example, an operator using a video monitor with a graphics 
overlay and a light pen or joystick night provide hunan interpretative assistance to the 
vision/wor Id modeling system. The operator night interactively assist the nodel matching 
algorithns by indicating with a light pen which features in the inage (e.g. edges, corners) 
correspond to those in a stored nodel. Alternatively, an operator could use a joystick to 
line up a wireframe model with a TV inage, either in 2-D or 3-D. The operator night either 
move the wireframe model so as to line up with the inage, or move the camera position so as 
to line up the image with the nodel. Once the alignment was nearly correct, the operator 
could allow automatic matching algorithms to complete the match, and track future movements 
of the image. 

2.5. Common Memory 

2 . 5 . 1. Communications 

One of the primary functions of common memory is to facilitate communications between 
modules. Communications within the control hierarchy is supported by a common memory in 
which state variables are globally defined. 
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Bach nodule in the sensory processing, world modeling, and task decomposition 
hlararchlas raads inpats from, and writes outputs to, the coaaon aeaory. Thus each module 
needs only to know where in coaaon aenory its input variables are stored, and where in 
coaaon aeaory it should write its output variables. The data structures in the coaaon 
aeaory then define the Interfaces between the G, N, and ■ nodules. 

The operator interfaces also interact with the systea through coaaon aeaory. The 
operator displays slaply read the variables they need froa the locations in coaaon aeaory. 
If the operator wishes to take control of the systea, he siaply writes coaaaad variables to 
the appropriate locations la coaaon aeaory. The control aodules that read froa those 
locations need not know whether their input conaands derived froa a huaan operator, or froa 
the next higher level in the autonoaous control hierarchy. 

2.5.2 State Variables 


The state variables in coaaon aeaory are the systea* s best estiaate of the state of the 
world, including both the external environaent and the internal state of the I, K, and G 
modules. Oats in coaaon aeaory are available to all aodules at all levels of the control 

systea. 

The knowledge base in the coaaon aeaory consists of three eleaentss asps which 
describe tne spatial occupancy of the world, object-attribute linked lists, and state 


3. LSVSLS XV THE COST SOL HIEBABCIY 

The control systea architecture described here for the Flight Telerobot Systea is a six 
level hierarchy as shown in Figure 3. At each level in this hierarchy a fundamental 
transf or nation is perforned on the task. 


Level 1 tranaforas coordinates froa a convenient coordinate fraae into joint coordinates. 
This level also servos joint positions, velocities, and forces. 

Level 2 coaputes inertial dynaaics, and generates saooth trajectories is a 

convenient coordinate fraae. 


Level 3 decoaposes elenentary aove conaands (B-aoves) into strings of interaediate poses. 

B-aoves are typically defined in tarns of notion of the subsysten being controlled 
(i.e., transporter, manipulator, camera platform, etc.) through a space defined by 
a convenient coordinate system. K-nove conaands nay consist of syabolic names of 
elementary noveaents, or nay be expressed as keyfraae descriptions of desired 
relationships to be achieved between systea state variables. 1- moves are 

decoaposed into strings of interaediate poses which define notion pathways that 
have been checked for clearance with potential obstacles, asd which avoid 
kineaatic singularities. 

Level 4 decoaposes object task commands specified in terns of actions perforned on objects 
into sequences of l-aoves defined in terns of manipulator notions. Object tasks 
typically define actions to be perforned by a single aultiarned telerobot systea 
on one object at a tine. Tasks defined in terns of actions on objects are 
decoaposed into sequences of B-aoves defined in terns of manipulator or vehicle 
subsystem notions. This decomposition checks to assure that there exist arotion 
freeways clear of obstacles between keyfrane poses, asd schedules coordinated 
activity of telerobot subsystems, such as the transporter, dual ara aanipulators , 
nul tif ingered grippers, and centra eras. 

Level 5 decoaposes actions to be perforned on batches of parts 
into tasks perforned on individual objects. It 

schedules the actions of one or aore telerobot systems 
to coordinate with other machines and systems operating 
in the inaediate vicinity. For example. Level 5 
decoaposes service bay action schedules into sequences 
of object task conaands to various telerobot servicers, 
astronauts, and autoaatic berthing aechaaisas. Service 
bay actions are typically specified in teras of 

servicing operations to be performed by all the systems 
(mechanical and huaan) in a service bay os a whole 
satellite. This decoaposi t ion typically assigns 
servicing tasks to various telerobot systems, and 
schedules servicing tasks so as to maximise the 

effectiveness of the service bay resources. 
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Level I a*co«poM« th« MttHitt ttrTiclB 9 alssion plan Into atrfica bay action couinoi. 

Mission plans are typically specified in tarns of satallita servicing priorities, 
raqairaaants * constraints, and Mission tins liaa. Tba level I daconpositlon 
typically assigns satallitas to service bays, sats priorities for service bay 
activities, generates raquiranants for spara parts and tool kits, and schadulas 
tba activities of tba service bays so as to aasinisa tba af f activaaass of tba 
satallita servicing Mission. To a larga axtant tba level € Mission plans will ba 
ganaratad off lina on tba ground, aithar by bnaan Mission plannars, or by 
autonatic or saniaotonatie Mission planning natbods. 


4 . DETAILSD STEDCTDBE OP TH B NO DO LX S 

Tba ■ nodule at each level consists of three parts as shown in Pigura 4s a job 
assignnant Manager JA, one or nor a planners PL(s), and one or noi - executors IX(s). 

Tba job assignnant Manager JA is responsible for partitioning tba task conn and TC into 
s spatially or logically distinct jobs to ba parfornad by s physically distinct 
plannar/axacutor nechanisas. At tba upper levels tba job assignnant nodule nay also assign 
physical resources against task elenents. Tba output of tba job assignnant Manager is a sat 
of job conaands JC(s), s«l, 2, ...« ■ where M is tba nunbar of spatially, or logically, 
distinct jobs. 

Por each of these job conaands JC(s), there exists a planner PL(s) and a executor 
BX(s). Bach planner PL(s) is responsible for decoaposing its job conaand JC(s} into a 
taaporal sequence of planned aubtaaks PST(s,tt). Planning typically raquiraa evaluation of 
alternative hypothetical sequences of planned subtasks. The planner bypotbasisas soae 
action or series of actions, the world nodal predicts tba results of the action (s) and 
conputas none evaluation function BP(s,tt) on tba predicted resulting state of tba world. 
Tba hypothetical sequence of actions producing tba bast evaluation function BP(s,tt)nax is 
then selected as tba plan PST(s,tt) to ba executed by tba executor BX(s). 

P5T (s , tt ) - PL (a) JC {*) ,SF(s, tt)aax 

where tt is the tine sequence index for steps in the plan. tt nay also be defined as a 
running tenporal index in planning space, tt * 1, 2, ..., th where tb is the value of tba 
tt index at the planning borison. The planning borixon is defined as the period into the 
future over which a plan is prepared. Bach level of the hierarchy has a planning borison of 
one or two expected input task tine durations. 

Bach executor BX(s> is responsible for successfully executing the plan PST(s,tt) 
prepared by its respective planner PL(s). If all the subtasks in the plan PST(s,tt) are 
successfully executed, then the goal cf the original task will be achieved. The executor 
operates by selecting a subtask fron the current queue of planned subtasks and outputting a 
subconnand STX(s,t) to the appropriate subordinate B nodule at tine t The EX (a) nodule 
Monitors its feedback FB(s,t) input in order to servo its output .* .(s,t) to the desired 
subtask activity. 


STX (s , tvn) - BUS) PST (s, t) ,PB (s, t) 

where n - the nunber of state clock periods required to conpute the function XX(s) . n 

typically equals 1. The feedback PB(s,t) also carries tining and subgoal event isfornation 
for coordination of output between executors at the sane level. when the executor detects e 
subgoal event, it selects the next planned subtask fron the queue. 

Executor output STX(s,t) also contains requests for infornation fron the world aodel M 
nodule, and status reports to the next higher (i+1) level in the B nodule hierarchy. The 
feedback PB(s.t) contains status reports fron the fl nodule at the i-1 th level indicating 
progress on its current task. 

5. CONCLUSION 

This paper has described a hierarchically organised control systee and has shown how 
this generic systen can be applied to telerobotic applications in space by considering the 
requlrenents of e flight telerobotic servicer for the space station. 
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TAILS 1 


OFBKATOB ZHTXBACTIOS AT BACH LBVSL 


LSTBL 

At th« servo 
sbove level 1 
above level 2 

above level 3 

above level 4 
above level 5 

above level I 


TTPB OF 1STBSACTI0S 

replica master, Individual joint position, rate, or force controllers, 
joy stick to perform resolved notion force/rate control 

indicate safe notion pathways. ftobot coaputes dynamically efficient 
movements 

graphically or syabolically define key poses* nenus to choose elemental 
moves. 

specify tasks to be performed on objects. 

reassign telerobots to different service bays. Insert, modify, and 
monitor plans describing servicing task sequences. 

reconfigure servicing mission priorities. 


161 



162 


FIGURE 1 s A llierarchial Control System Architecture for Intelligent Vehicles 



Task Decomposition 





ACTUATORS 


FIGURE 3: A six level Kierarchial Control Systcr. Proposed for 

Multiple Autonomous Vehicles 
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